Augmented reality therapeutic movement display and gesture analyzer

ABSTRACT

Systems and methods for displaying augmented reality clinical movements may use an augmented reality device to display aspects of a clinical movement. The systems and methods may use a motion capture device to capture the clinical movement. A method may include analyzing information about the clinical movement to determine a path of motion representative of at least a portion of the clinical movement. The method may automatically define a path region or a virtual target in an augmented reality environment overlaid on a real environment. The method may display the path region or the virtual target on an augmented reality display.

CLAIM OF PRIORITY

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/416,869, filed on Nov. 3, 2016, and claims thebenefit of U.S. Provisional Patent Application Ser. No. 62/421,001,filed on Nov. 11, 2016, and claims the benefit of U.S. ProvisionalPatent Application Ser. No. 62/440,588, filed on Dec. 30, 2016, thebenefit of priority of each of which is claimed hereby, and each ofwhich are incorporated by reference herein in its entirety.

BACKGROUND

Telerehabilitation systems are typically used to remotely assess ormonitor patients engaged in rehabilitation activities. Currenttelerehabilitation systems are often limited or not used foroccupational or physical therapy due to the remote nature oftelerehabilitation. Occupational or physical therapy includes exercisesor activities to recover from an injury, surgery, or to otherwiseimprove mobility. Often, patients forget how to do activities associatedwith rehabilitation or are unable to understand the providedinstructions describing the activities.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments discussed in the presentdocument.

FIG. 1 illustrates real and virtual aspects of an augmented reality andexercise creation system in accordance with some embodiments.

FIG. 2 illustrates an augmented reality display in accordance with someembodiments.

FIG. 3 illustrates an automatic gesture display system in accordancewith some embodiments.

FIG. 4 illustrates a patient evaluation display system in accordancewith some embodiments.

FIG. 5 illustrates a telerehabilitation system in accordance with someembodiments.

FIG. 6 illustrates a flow chart showing a technique fortelerehabilitation in accordance with some embodiments.

FIG. 7 illustrates a flow chart showing a technique for displayingdirections related to a therapeutic movement for a patient within anaugmented reality environment in accordance with some embodiments.

FIG. 8 illustrates a flow chart showing a technique for displayingaugmented reality clinical movements in accordance with someembodiments.

FIG. 9 illustrates generally an example of a block diagram of a machineupon which any one or more of the techniques discussed herein mayperform in accordance with some embodiments.

DETAILED DESCRIPTION

Systems and methods for telerehabilitation feedback are describedherein. The systems and methods herein describe using a movement captureapparatus to capture telerehabilitation information from a therapist orpatient. The telerehabilitation systems and methods may use a userinterface or video display, an augmented reality (AR) display, a virtualreality (VR) display, audible alerts, haptic feedback, a non-contactalert, or the like to present feedback.

In an example, a telerehabilitation system may be used to provide amechanism that captures video of a rehabilitation exercise andautomatically determines key points on a therapist performing theexercise in the video. The key points may be used to create visualtargets, of the exercise, for a patient when the patient is attemptingto perform the exercise. The system may demonstrate a rehabilitationexercise virtually, including the visual targets. The visual targets mayinclude a “bubble,” such as a spherical or circular visualrepresentation that may be “popped” (e.g., by showing an animation orvisual effect removing the bubble from display when the patient or anobject interacts with the bubble). The system may capture video of thepatient performing the exercise, and may analyze the patient's attempt.The visual targets may be used to display the effectiveness of thepatient's attempt, for example popped bubbles may represent a successfulcompletion of the exercise or parts of the exercise. The system mayprovide feedback to the patient (e.g., the popped bubbles or un-poppedbubbles).

The systems and methods described herein may automatically determine aplacement for a gesture bubble (e.g. a bubble placed for a gesture, suchas a beginning of a gesture or an end of a gesture for an exercise),such as based on therapist movements. For example, a therapist may standin front of a movement capture apparatus (e.g., a sensor, a sensorarray, a camera, an infrared camera, two or more cameras, a depthcamera, etc.) and perform a motion, and the system may automaticallyplace a gesture bubble. The system may place the bubble in thedetermined location, and proceed to a second bubble. For example, afirst bubble may correspond with a first location (e.g., a startinglocation), and a second bubble may correspond with a second location(e.g., an ending location). A path region may include a path between thefirst location and the second location, or may include the startinglocation or the ending location. In an example, a gesture video may becreated using the path region, one or more bubbles, captured video froma therapist, captured video from a patient, an animation segment, or thelike. In an example, a gesture may include a movement in an exercise, ormay be an exercise (e.g., a gesture may include a movement with multiplemovements making up an exercise, or a gesture may include a movement ofan exercise, which may be repeated or include other gestures orexercises to form a routine).

Augmented reality (AR) is a technology for displaying virtual or“augmented” objects or visual effects overlaid on a real environment.The real environment may include a room or specific area, or may be moregeneral to include the world at large. The virtual aspects overlaid onthe real environment may be represented as anchored or in a set positionrelative to one or more aspects of the real environment. For example, avirtual object may be configured to appear to be resting on a table. AnAR system may present virtual aspects that are fixed to a real objectwithout regard to a perspective of a viewer or viewers of the AR system.For example, a virtual object may exist in a room, visible to a viewerof the AR system within the room and not visible to a viewer of the ARsystem outside the room. The virtual object in the room may be displayedto the viewer outside the room when the viewer enters the room. In thisexample, the room may act as a real object that the virtual object isfixed to in the AR system.

An AR system may be viewable to one or more viewers, and may includedifferences among views available for the one or more viewers whileretaining some aspects as universal among the views. For example, aheads-up display may change between two views while virtual objects maybe fixed to a real object or area in both views. Aspects such as a colorof an object, lighting, or other changes may be made among the viewswithout changing a fixed position of at least one virtual object.

A user may see a virtual object presented in an AR system as opaque oras including some level of transparency. In an example, the user mayinteract with the virtual object, such as by moving the virtual objectfrom a first position to a second position. For example, the user maymove an object with his or her hand. This may be done in the AR systemvirtually by determining that the hand has moved into a positioncoincident or adjacent to the virtual object (e.g., using one or morecameras), and causing the virtual object to move in response. Virtualaspects may include virtual representations of real world objects or mayinclude visual effects, such as lighting effects, etc. The AR system mayinclude rules to govern the behavior of virtual objects, such assubjecting a virtual object to gravity or friction, or may include otherpredefined rules that defy real world physical constraints (e.g.,floating objects, perpetual motion, etc.).

Systems and methods for augmented reality (AR) telerehabilitation aredescribed herein. The systems and methods herein describe using AR todisplay a virtual target for a patient to allow the patient to completea clinical movement. In an example, a virtual therapist may bedisplayed. In another example, a real therapist may be displayed with anAR augmentation, such as highlighting of a body part, virtualdirections, etc. The virtual therapist or the real therapist maydemonstrate the clinical movement. The clinical movement may include theuse of an object. In an example, the AR may be used to display objectsto be used by the patient. In an example, the AR may be used to displaydirections for a patient to perform a clinical movement. The realtherapist may, for example, be a live therapist appearing with apatient, or may be a video recording of a therapist. The real therapistmay be defined as any therapist that is not virtual.

The systems and methods described herein may provide overlaid virtualgraphics to assist in a user's understanding of how to perform amovement. A path or target may be highlighted, a virtual representationof a patient or therapist may be shown performing the movement (e.g.,overlaid on the user—as a point-of-view technique to show the movement).A virtual target may be used to aid a patient in visualizing a range tothe movement (e.g., an ultimate end point for the movement).

FIG. 1 illustrates real and virtual aspects of an augmented reality andexercise creation system 100 in accordance with some embodiments. In anexample, the system 100 includes a video capture device 105, a displaydevice 107, and an input device 109 that can be used by the therapist102 to generate model exercises for a rehabilitation patient. Thetherapist 102 may perform an exercise, which is captured by the videocapture device 105 and displayed in the display device 107. The inputdevice 109 may be used to edit or augment the displayed exercise, or toselect one or more exercises for a routine. The system 100 mayautomatically edit the captured video to remove extraneous portions thatcome before or after the exercise performed by the therapist 102. In anexample, a series of exercises may be performed by the therapist 102 andcaptured by the video capture device 105, and the system 100 may splitthe captured video of the series of exercises into individual exercisevideos. In an example, the video capture device 105 may be a Kinect fromMicrosoft of Redmond, Wash.

Aspects of the exercise may be selected using the input device 109. Theselected aspects may include a starting position, an ending position, ora transition motion. When a starting position is selected, the displaydevice 107 may display the selection at the appropriate time in thecaptured video of the exercise. For example, a circle may be drawnaround a displayed body part (e.g., a foot, a hand, etc.), which may bedisplayed in the captured video for the exercise. Similarly, an endingposition may be highlighted. When a transition motion is selected, apath may be displayed during the captured video that tracks with theselection. The starting position, ending position, or the transitionmotion may include more area on the captured video than the body partoccupies (e.g., a radius around a center point of the body part).

In an example, the system 100 may be calibrated using the video capturedevice 105. The video capture device 105 may use infrared light todetect the therapist 102 in a field of view. The system 100 may evaluatethe detection to identify joints, limbs, appendages, a head, etc., ofthe therapist 102. These identified body parts may be used with latercaptured video of an exercise to label specific body parts.

After a video of an exercise is captured by the video capture device105, the therapist 102 (or another user) may edit the captured video. Inan example, the therapist 102 may select portions of a captured videoand add tags, such as “introduction,” “exercise,” “first repetition,”“second repetition,” “outro,” or the like. In an example, a singlerepetition captured may be repeated in an edited video to show multiplerepetitions for patient viewing.

A final edited video may be created for an exercise. The final editedvideo may be named and given a category tag, such as a body part, amuscle group, a post-surgery type designation, a patient-specific tag,or the like. The final edited video may be saved for later use inconstructing a routine, such as by the therapist 102. In anotherexample, the final edited video may be saved to a database to be sharedwith other users (e.g., other users caring for a patient shared with thetherapist 102, other therapists in a company, group, or hospital,publicly, or the like). In an example, the system 100 may be used torepeat an exercise a specified number of times so that a patient mayview the exercise the specified number of times to complete a routine,or part of a routine.

The system 100 may be used to capture three-dimensional movement. Forexample, the video capture device 105 may include a movement captureapparatus. The movement capture apparatus may include two or moreinfrared sensors or cameras to detect or capture three dimensionalmovement. The video capture device 105 may include a camera to capturevideo in conjunction with infrared captured movement. The movementcaptured may include video.

The system 100 may interact with a real therapist (e.g., therapist 102),or may include a virtual therapist 103 displayed within the system 100.The system 100 may be used by a patient 104. The system 100 includes anAR device 108. In an example, the system 100 may include a referencewall 106, a physical object 112, a virtual object 116, or a camera 110.In an example, the physical object 112 may include a sensor 114. Inanother example, a sensor may be embedded on an implant of the patient104. In an example, the camera may be coupled to the AR device 108. TheAR device 108 may include a headset, glasses, goggles, contacts, aprojector, or the like. In an example, the reference wall 106 may beidentified during configuration or calibration of the AR device 108.

The AR device 108 may include the camera 110. The camera 110 may includean infrared camera, an infrared filter, a visible light filter, aplurality of cameras, a depth camera, etc. The AR device 108 may projectvirtual items over a representation of a real environment, which may beviewed by the patient 104. In an example, the real environment mayinclude display of a floor, a room, and physical props, among otherthings. The therapist 102 may be present in the real environment, andvirtual aspects may be overlaid on the therapist 102 within the ARenvironment generated by the AR device 108. In another example, thevirtual therapist 103 may include virtual aspects, such as highlightingof body parts, movement arrows, blurring to show motion, or the like.The virtual therapist 103 may be placed in the real environment at apredetermined location, such as a location in the real environment thatis within an area visible to the patient 104 or the AR device 108. Forexample, the virtual therapist 103 may located in front of the AR device108 within the system 100 as visible by the patient 104, such that thepatient 104 may view the virtual therapist 103. When the patient 104turns the AR device 108, the virtual therapist 103 may be designed todisappear from view similar to how the real therapist 102 woulddisappear if the patient's head turned.

In an example, the virtual therapist 103 may be shown overlaid on thereal environment, to demonstrate an exercise. In an example, a virtualdisplay of props or objects, such as for use in exercise may be overlaidon the real environment. For example, the virtual object 116 mayrepresent the physical object 112. The AR device 108 may be used toinstruct the patient 104 to find the physical object 112 in the room bydisplaying the virtual object 116 as an example. In an example, thevirtual object 116 may be displayed in use by the virtual therapist 103or may be displayed adjacent to, above, or floating near, the realtherapist 102.

The physical object 112 may include the sensor 114. The sensor 114 maybe used to track patient progress, such as a duration or number ofrepetitions completed by the patient 104. In an example, the sensor 114may be used to identify the physical object 112 to the AR device 108.Once identified, the AR device 108 may select the virtual object 116that corresponds to the physical object 112. For example, if the patient104 has a ten pound weight and a five pound weight available, these maybe identified via sensors in the weights, and the AR device 108 maydetermine the five pound weight is to be used in an exercise, and thevirtual object 116 may resemble the five pound weight. The virtualobject 116 may change to resemble the ten pound weight in anotherexercise or when increasing weight to increase difficulty for theexercise. In an example, the AR device 108 may display a virtualmovement of a limb or a virtual exercise or may display virtual bubblesfor indicating a starting position, an ending position, or a path for amovement or exercise.

In an example, an implant sensor may be embedded in an implant in thepatient 104. The implant sensor may be used to track movement (such as anumber of repetitions), non-movement, etc., of the patient 104. Thistracked movement may be used to augment movement captured by the camera110 or a patient-facing movement capture apparatus (e.g., a sensor, asensor array, a camera, an infrared camera, two or more cameras, a depthcamera, etc.). Movement of the patient 104 may be tracked using apatient-facing movement capture apparatus, the camera 110, the implantsensor, or any of the movement capture apparatus, among other things. Inan example, therapy output or therapy prescribed may be changed orupdated based on sensor data from the implant sensor. For example, witha total knee arthroplasty the knee prosthesis (implant) may include asensor to monitor pressure generated during movement, and when there istoo much pressure on one side of an implant, an indication to ease themovement, change the movement, or stop the movement may be displayedusing the AR device 108. The implant sensor or AR device 108 data (e.g.,whether the patient 104 is successfully performing movements, apercentage of successful performance, metrics related to number ofrepetitions, weight used, etc., or the like) may be used by thetherapist 102 or other caregiver (e.g., a surgeon) after the patient 104performs a movement, completes a scheduled session, or performs at leastpart of scheduled therapy. The data may be used to adjust prescribedtherapy, movement, exercises, medication, surgery timetable, or thelike. The patient 104 and the real therapist 102 may be able to use theAR device 108 or multiple AR devices to share an augmented experience.For example, the real therapist 102 may have an AR device, and the ARdevice of the therapist 102 may display augmented and virtual aspects ina shared environment with the AR device 108, overlaid on the same realenvironment. The real therapist 102 may manipulate virtual or realaspects of the shared environment such that the patient 104 may see themanipulation. For example, the real therapist 102 may pop a virtualbubble or lift a virtual object, wherein the virtual bubble or thevirtual object may be visible to the patient 104 and the real therapist102, as further discussed below with respect to FIG. 2 and as describedherein below in paragraph and elsewhere.

FIG. 2 illustrates an augmented reality (AR) display 200 in accordancewith some embodiments. The AR display 200 may be used by a patient 201,to display virtual aspects in a real environment. The AR display 200 mayinclude a virtual identification of a joint 202 of the patient 201, areference wall 218 displayed in the real environment, or a plurality ofvirtual reference indications. The reference indications may include avirtual starting bubble 206, a virtual ending bubble 210, a virtual pathof motion 208, virtual edges 214 and 216 to a path region, a virtualstarting limb position 204, or a virtual ending limb position 212. In anexample, the reference indications may include a virtual examplestarting bubble 220, a virtual example ending bubble 222, or a virtualexample path region 224. The virtual example starting bubble 220, thevirtual example ending bubble 222, or the virtual example path region224 may be displayed overlaid within the AR environment on arepresentation of a therapist present in the real environment. Inanother example, the virtual example starting bubble 220, the virtualexample ending bubble 222, or the virtual example path region 224 may bedisplayed overlaid on a virtual therapist. In yet another example, thevirtual example starting bubble 220, the virtual example ending bubble222, or the virtual example path region 224 may be displayed with avirtual limb (e.g., a body part of a virtual therapist withoutdisplaying an entire virtual therapist). In another example, the virtualexample starting bubble 220, the virtual example ending bubble 222, orthe virtual example path region 224 may be displayed without a virtualor without a therapist present in the real environment.

In an example, the virtual starting bubble 206, the virtual endingbubble 210, the virtual path of motion 208, or the virtual edges 214 or216 to the path region may be displayed virtually. The virtual startingbubble 206, the virtual ending bubble 210, the virtual path of motion208, or the virtual edges 214 or 216 to the path region may be displayedusing a color or a plurality of changing colors, may be sequentiallydisplayed, or the like. In an example, the virtual starting bubble 206,the virtual ending bubble 210, the virtual path of motion 208, or thevirtual edges 214 or 216 to the path region may be “popped” or otherwiseremoved from view if the patient performs a movement corresponding tothe virtual representations. For example, if the patient physicallyplaces a body part in a location occupied in the AR by the virtualstarting bubble 206, the virtual starting bubble 206 may pop. In anexample, the virtual path of motion 208 may include a plurality ofbubbles to be popped as a body part moves from the virtual startingbubble 206 to the virtual ending bubble 210.

In an example, an AR device may be used to generate the AR display 200.The AR device may include a projection screen, goggles, glasses, etc. Inan example, the AR device may project an animation around the patient201, allowing the patient 201 to see the virtual starting bubble 206,the virtual ending bubble 210, the virtual path of motion 208, thevirtual edges 214 and 216 to the path region, the virtual starting limbposition 204, or the virtual ending limb position 212. When the patient201 moves, the AR device may present an avatar, such as a virtualtherapist, to present a virtual therapy experience. The virtualtherapist may be preprogrammed or controlled in real time by atherapist. In an example, the patient 201 may have an opportunity to askquestions of the virtual therapist using the AR display 200 or inputdevices within the AR device 108. Therapy for the patient 201 may bechanged based on the questions, answers, or interactions with thepatient 201. For example, based on how the patient 201 interacts,exercises may decrease or increase in difficulty or duration. In anexample, the patient 201 may ask the therapist to repeat or describe anexercise. The avatar may direct the patient 201 to a physical location,model an exercise, offer encouragement, correction, modification, showsuccess or failure, or the like. In an example, the avatar may occupy avirtual physical presence, such as a static position in a room (e.g.,relative to the reference wall 218).

The AR display 200 may allow the patient 201 to select a joint orlocation on an AR avatar. By selecting a joint or location on the ARavatar, the patient 201 may indicate what hurts or needs attention onthe patient 201, select a side of the body, be prompted for a painlevel, or the like. In another example, the patient 201 may selectoptions from a user interface within the AR display 200, such as, ascale bar to select pain. The AR display 200 may include a questionnairefor the patient to answer so that a therapist may evaluate progress ordetermine exercises for the patient.

In an example, the patient 201 may have a personalized movement profile.The personalized movement profile may be applied by an AR device tocause the AR display 200 to be personalized. The personalized AR display200 may include specific exercises, movements, limitations, or the likethat are custom to the patient 201. For example, if the patient 201 hasreceived an implant, such as received during a total knee arthroscopy,hip replacement, heart device implant surgery, etc., the AR display 200may include exercises designed with these implants or surgeries in mind.The AR display 200 may include education for the patient 201, such as toimprove outcomes or mitigate movement that may hurt the patient.

In an example, the AR display 200 may include a virtual mirror. Thevirtual mirror may be used in a mirrorboxing technique. Mirrorboxing maybe used to mirror a limb, such as when a mirror limb is not present. Forexample, if the patient 201 has lost the left leg below the knee, theright leg may be mirrored in the AR display 200, such that the ARdisplay 200 may display the left leg below the knee by mirroring theright leg. In another example, the left leg below the knee may bedisplayed virtually, based on the right leg, or constructed from virtualcomponents. Mirrorboxing may be used to give the patient 201 relief fromphantom limb pain, allow a patient to attempt to regain use of a limb,relieve pain in a limb, etc. The virtually displayed limb may be used tomap movement in the brain of the patient 201, such as by superimposingthe limb and instructing the patient 201 to move the limb virtually. Forexample, the patient 201 may not have control of the right arm and maymove the left arm. When the left arm moves in the real environment, thatmay cause the right arm to move virtually in the AR display 200. Thebrain of the patient 201 may map the pathway to actually control theright arm by moving the left arm and viewing the right arm moving. Thistechnique may be used to heal or repair the movement ability of theright arm.

FIG. 3 illustrates an automatic gesture display system 300 in accordancewith some embodiments. The automatic gesture display system 300 includesa user interface 302. The user interface 302 includes a video/animationautomatic gesture display component 304. The video/animation automaticgesture display component 304 may be used to add movement bubbles (e.g.,308 or 314) or a path region, manually or automatically. In an example,movement bubbles (e.g., 308 or 314) may be added to a path of motion 312automatically. For example, including success parameters or failureparameters may be added to the path of motion 212 to create a pathregion (e.g., a region around the path of motion 312 between edge 318and edge 320). The path region may be used to show a patient how toperform the clinical movement. The path region may be used to determinewhether the movement by the patient was successful. In an example, thepath region or the path motion may be displayed on a display (e.g., ascreen, an AR display, a VR display, etc.).

In an example, the path region may be used to determine whether apatient has successfully completed an exercise. For example, if thepatient completes the exercise within the path region, the exercise maybe determined to be completed successfully. If the patient moves outsidethe path region while attempting to complete the exercise, the exercisemay be determined to not be completed successfully. In another example,the movement bubbles may be used to analyze a patient video to determineif the patient has properly performed the exercise. For example, if thepatient is able to pop the bubbles (e.g., all the bubbles in anexercise), exercise while performing an exercise, the exercise may bedetermined to be completed successfully. In an example, the exercise maybe partially completed if one or more bubbles are popped. In anotherexample, the path region and the bubbles may be used to determine if theexercise has been completed successfully.

The path region may be represented in two dimensions or threedimensions. For example, the path region may include a two dimensionexpansion of the path of motion 312. In another example, the path regionmay include a three dimension expansion of the path of motion 312. Forexample, the path region may include an area within a radial distance(e.g., in two or three dimensions) away from the path of motion 312. Inan example, a starting position or an ending position may be indicatedfor the path of motion 312. The starting position or the ending positionmay include a radial distance (e.g., two or three dimensional) away froma starting point or ending point of the path of motion 312 respectively.For example, the starting position may include a circle or sphere aroundthe starting point of the path of motion 312. The starting point may beindicated by a therapist or may be detected automatically.

The video/animation automatic gesture display component 304 includes astart bubble 308 at a starting location (e.g., corresponding to thestarting location 208 of FIG. 2) and an end bubble 314 at an endinglocation (e.g., corresponding to the ending location 214 of FIG. 2). Thevideo/animation automatic gesture display component 304 includes acaptured video displayed or animated joint 306, and limb locations 310and 316 corresponding to a start and an end respectively. Thevideo/animation automatic gesture display component 304 mayautomatically generate the start bubble 308, the end bubble 314, theedge 318, or the edge 320. In an example, the edges 318 and 320 may beautomatically generated at a predetermined distance from the path ofmotion 312 (e.g., a number of pixels, a distance determined using ascaling of captured video or an animation distance).

In an example, a clinical movement captured from a therapist may belater animated and the animation may be used with a path region tocreate a gesture video. In another example, a therapist may select apreviously generated animation to create a gesture video. In anotherexample, a therapist may select a previously captured video to create agesture video. To create a gesture video, a therapist may select a jointthat moves (e.g., joint 306), for example a knee joint, a hip joint, anelbow joint, a shoulder joint, a neck joint, etc. The joint 306 may beautomatically detected in a captured video, and a path region may beapplied to movement extending from the joint 306 (e.g., if a knee jointis selected, the path region may be for the foot as it travels while theknee joint is extended, or, in another example, the selected joint maybe used as the path region), such as along the path of motion 312between the edges 318 and 320 from the start bubble 308 to the endbubble 314.

In an example, a movement capture apparatus may be used to capture videoand the start bubble 308 and the end bubble 314 may be automaticallyadded to the captured video. The start bubble 308 or the end bubble 314may be color coordinated, such as a green start bubble 308 indicatingthe starting point for a movement and a red end bubble 314 indicatingthe ending point for the movement. In an example, the colors may changeas the movement is in progress. For example, the start bubble 308 may bea first color and then change as a specific body part or object isplaced within the start bubble 308, upon which the start bubble 308 maychange to a second color to indicate the placement is correct. The edges318 or 320 may similarly change color based on whether movement iswithin or outside the edges 318 or 320. The end bubble 314 may similarlychange color when the body part or object is placed in the end bubble314. In an example, the start bubble 308 or the end bubble 314 may be“popped” (e.g., animated to disappear), such as when a user places abody part or object (e.g., a predetermined specific body part or object)within the start bubble 308 or the end bubble 314. In an example, aseries of bubbles may be placed between the start bubble 308 and the endbubble 314, such as along the path of motion 312. The series of bubblesmay change color or pop as described above for the start bubble 308 orthe end bubble 314. In another example, the start bubble 308 and the endbubble 314 may be in the same place or near each other. For example, thepath of motion 312 may include a round trip or movement away from thestart bubble 308 and then back towards or near the start bubble 308,which may transition during the movement to being the end bubble 314.

In an example, a therapist may select a gesture flag to identify jointsthat the patient must move in order to satisfy the exercise conditions.A movement capture apparatus may be used to determine whether the motionis captured and whether the identified joints are moved in the wayindicated by the gesture flag(s). For example, a gesture flag selectedby the therapist may correspond to a joint tag identified by themovement capture apparatus.

In an example, bubbles may be used to indicate a restriction onmovement. For example, a particular movement may include a limit on headmovement to within a certain bubble to ensure that the patient correctlyperforms the movement.

The user interface 302 of FIG. 3 may be used as a video/animationcreation display system in accordance with some embodiments. The userinterface 302 may include a video/animation creation component. Themovement may be done along a path of motion 312. The video/animationcreation component may be used to edit or display captured video, editor display animation, or edit or display a location or a path of motion,such as an automatically generated location or path of motion. In anexample, a therapist may record a clinical movement using a movementcapture apparatus, such as an infrared sensor or a camera. The recordingmay use a plurality of cameras. The recorded clinical movement may beanalyzed to determine the path of motion 312.

The automatic gesture display system 300 may include an AR authoringtool. The AR authoring tool may be used to augment aspects of a detectedgesture or clinical movement. For example, the AR authoring tool may beused to change gestures or movements. In an example, the AR authoringtool may be used to create multiple different views for a gesture ormovement. In another example, the AR authoring tool may be used toaugment parts of a path of motion or a target object. For example, thepath of motion may be augmented with a color, a distance from a centerof the path of motion may be selected (e.g., to expand or contract aregion around the path of motion), or set waypoints along the path ofmotion to indicate or identify progress along the path of motion. Thetarget object may be augmented with a shape, color, style (e.g.,flashing, pulsing, shimmering, or the like), transparency, etc.

In an example, an augmented reality device is used by a patient to see athree-dimensional animation rendered in an augmented reality display.For example, the animation may include a virtual representation of atherapist performing a clinical movement in an augmented realityenvironment overlaid on a real environment. In another example, theanimation may be a virtual reality animation. In another example, theanimation may be an augmented animation enhancing a clinical movementperformed by a therapist in front of the patient in a real environment.For example, the therapist's arm or leg or other movement limb may beaugmented, such as with a color, outline, arrow, etc., and enhancedwhile the therapist performs a clinical movement.

In an example, creating a path of motion or a target object may beautomated with a target range of motion, such as one based on adiagnosis. For example, a therapist may perform a complete clinicalmovement, which may include one or more waypoints along the completeclinical movement, the waypoints representing progress points. Thewaypoints may include an initial partial clinical movement correspondingwith a first target object along the path of motion, a second partialclinical movement corresponding with a second target object along thepath of motion, etc. An ultimate target may be placed at the end of thepath of motion. In this way, a plurality of path of motion targets maybe established representing a full range of motion. The full range ofmotion may be broken down into partial range of motion segments, whichmay be displayed (e.g., progressively) to a patient in an augmentedreality environment. The segments may be coupled to increase the rangeof motion targets progressively (e.g., each day adding another targetwith a wider range of motion). The range of motion may includefunctional measures that may be achieved by a patient. In an example,the range of motion may be changed based on a patient's height, weight,range of motion, proportionality, etc.

In an example, creating a path of motion or a target object may includeautomation, such as based on expected experiences at home, in theclinic, at work, etc. For example, a pet, slippery rug, or otheractivities performed by a patient in daily life may be automaticallyadded to the augmented reality environment to reflect everydayactivities in a therapeutic setting. These expected experiences mayinclude canned protocols that may be manipulated individually, or may befrom a database of common activities.

In an example, complex paths may be created by a therapist for apatient. For example, multi-step movements may be created with specificwaypoints for the patient to stop at during the movements. For example,a complex path may include a first path ending at a first waypoint wherethe patient raises an arm up 90 degrees, and then a second path endingat a second waypoint where the patient moves the arm 90 degrees out.Paths may be created separately and added together by the therapist tocreate a complex path, or a complex path may be created as a step bystep process.

FIG. 4 illustrates a patient evaluation display system 400 in accordancewith some embodiments. The patient evaluation display system 400includes a user interface 402. In an example, the user interfaces 202,302, and 402 may be a single user interface with different views. Inanother example, the user interfaces 202, 302, and 402 may be accessedusing credentials, the credentials allowing access to one or more of theuser interfaces 202, 302, and 402, and optionally denying access to oneor more of the user interfaces 202, 302, or 402. The user interface 402includes a video/animation evaluation component 404. The video/animationevaluation component 404 includes a joint 406 of a patient (e.g., usingcaptured video of a patient or a live recording of a patient), and limblocations at a start 410 and an end 416. The video/animation evaluationcomponent 404 includes an actual path the patient performed, with anactual starting location 408, an actual ending location 414, and anactual path of motion 412. The video/animation evaluation component 404includes an intended path for the patient, with an intended startinglocation 422 (e.g., the start bubble 308 of FIG. 3), an intended endinglocation 426 (e.g., the end bubble 314 of FIG. 3), and an intended pathof motion 424 (e.g., the path of motion 312). The intended path mayinclude intended edges 418 and 420 of an intended path region. In anexample, if the actual path of motion 412 falls between the intendededges 418 and 420, the patient's attempt at a clinical movement may bedetermined to be successful. In another example, if the actual path ofmotion 412 falls outside the intended edges 418 and 420, the patient'sattempt at a clinical movement may be determined to be a failure. Inanother example, some amount of error may be tolerated, such as a briefmovement outside the intended edges 418 and 420.

In an example, the actual starting location 408 may be compared with theintended starting location 422. If the actual starting location 408aligns with the intended starting location 422, falls within theintended starting location 422, overlaps the intended starting location422, etc., the actual starting location 408 may be determined to besuccessful. A similar alignment determination may be made for the actualending location 414 and the intended ending location 426.

With a gesture video, either previously created and selected, or a newlycreated gesture video, a corresponding path region (e.g., enclosed byedges 418 and 420) may be determined for a patient. For example, a videoof a therapist moving a joint may include a starting location, a pathregion, and an ending location. The therapist may differ in size from apatient, and the starting location, the path region, or the endinglocation may be automatically adjusted for the patient to fit the sizeof the patient. The starting location, the path region, and the endinglocation may be converted into the intended starting location 422, theintended path of motion 424, and the intended ending location 426respectively, such as using the therapist size or the patient size. Forexample, if the therapist is short and the patient is tall, and thevideo shows the therapist raising an arm overhead, the patient's arm mayraise to a higher height. The ending location may be moved to thishigher height automatically based on detecting the patient's size. Thetherapist's size may be automatically detected, and may be stored withthe video. In an example, the size of the starting location, the pathregion, or the ending location may be changed. These changes tolocations or sizes may be done automatically and may be proportionalchanges such that the motion to be done by the patient is similar to themotion done by the therapist.

In an example, the patient evaluation display system 400 may be used toautomatically detect or identify an orientation of a patient to amovement capture apparatus. The orientation may be compared to anorientation of captured video or animation used to display an exercise.For example, the patient may be instructed using the user interface 402to turn a specified number of degrees, sit down, stand up, etc., so thatthe patient is in a correct starting position. In another example, oneor more starting bubbles (e.g., the intended starting location 422) maybe used to direct the patient to a starting position. For example, theintended starting location 422 may be used as an initial placement for abody part to be moved during the exercise. Additional starting locationsmay be used, such as a head location, torso location, leg location, armlocation, or the like, or a visual indication such as a directionalarrow may be displayed to provide a starting orientation for the patientto begin the exercise. In another example, the orientation of the patentmay be identified and the displayed video or animation may be rotated tocorrespond with the orientation of the patient.

In an example, the intended starting location 422, the intended path ofmotion 424, the intended ending location 426, or other bubbles may bechanged automatically. For example, the bubbles may be changed to createa next level, such as to increase stress, increase challenge for thepatient (e.g., by moving the bubbles further away from patient, changingthe exercise, etc.). The automatic adjustments may be made based on aprogression, such as a progression preselected by a therapist. In anexample, a therapist may select a starting point and an ending point,and the video/animation evaluation component 404 may automaticallyinterpolate points in between to adjust bubbles to change the way thatthe patient proceeds. For example, the progression may be based on aunique starting point of the patient's current success or currentmovement pattern (e.g., level of activity), and then may automaticallycreate the intended path of motion 424 to get to a patient's uniquefinish goal or intermediate goal. In an example, difficulty of intendedmotion may be changed, for example by changing position of the bubbles,changing bubble sizes, changing the angle between the intended startinglocation 408 and the intended ending location 426 from the joint 406, orthe like.

In an example, the video/animation evaluation component 404 may showvideo captured or live of a patient performing a movement. The videocaptured may show reps done by the patient (e.g., automaticallydetermined based on the bubbles being activated, reached, popped, etc.),duration, heart rate, or the like. The video capture may include askeleton of the patient or the patient, and may black out any otherbackground. In another example, a patient may self-report reps orduration.

In an example, the augmented reality device 501 may be used to displayone or more previous attempts by a patient at performing a clinicalmovement. For example, the camera 506 may capture the patient performingthe clinical movement for a first time, including a first range ofmotion (e.g., up to a first target object or a first patient-specificwaypoint). The first clinical movement attempt may be stored in thememory 504 or the database 511. Then, for example when the patientattempts the clinical movement a second or later time, the first attemptmay be shown to the patient in the augmented reality display 510. Theone or more previous attempts by the patient may be shown with aparticular effect, such as a ghosting effect (e.g., faded, dim, orethereal). The previous attempts may be shown in the augmented realitydisplay 510 as the patient attempts the clinical movement at the latertime, (e.g., previous attempts shown in real-time with the patientattempting the clinical movement). In another example, the previousattempts may be shown on the display 510 to a therapist to show progressby a patient. In an example, attempts that are more remote in time tothe present time may be shown fainter. In another example, the previousattempts may be color coded or numbered. In an example, a before andafter overlay may be used by the therapist to display to the patient theprogress that the patient is making with range of motion on the clinicalmovement. The previous attempts may be shown to the patient using afirst person view, displaying a range of motion with progression thatmay be personalized to the patient.

FIG. 5 illustrates a system 500 for displaying augmented realityclinical movements in accordance with some embodiments. The system 500includes a gesture analyzer device 503. The gesture analyzer device 503may include a processor and memory or may be connected to a device, suchas the augmented reality device 501 that includes a processor 502 andmemory 504. In an example, the gesture analyzer device 503 may include amovement capture apparatus (e.g., a camera or a Kinect) 503. Theaugmented reality device 501 may include a feedback controller 508 or adisplay 510. The gesture analyzer device 503 may be in communicationwith a database 511. The database 511 may include video storage 512 oranimation storage 514. In an example, the augmented reality device 501may be a Holo Lens manufactured by Microsoft of Redmond, Wash.

The processor 502 may be used to receive information about a clinicalmovement of a therapist, such as a clinical movement captured using themovement capture apparatus 503. The processor 502 may analyze theclinical movement to determine a path of motion of the clinicalmovement, such as a path of motion on video captured by the movementcapture apparatus 503. The processor 502 may automatically define a pathregion, such as by using the path of motion. The processor 502 mayreceive information about a movement of a patient along the path ofmotion, such as movement of the patient captured using the movementcapture apparatus 503. The processor 502 may determine whether themovement was within the path region. In an example, the processor 502may send feedback, such as to the feedback controller 508 or the display510. The feedback may indicate whether the movement was within the pathregion. The display 510 may display the feedback, such as by visuallyindicating (e.g., on a user interface) whether the movement was withinor outside the path region or where the movement may have been outsidethe path region. The feedback controller 508 may be used to send thefeedback to the display 510, issue an audible alert, provide hapticfeedback, or the like. In an example, the display 510 may be a screen,an augmented reality display, a virtual reality display, or the like.

The processor 502 may determine a start position or an end positionautomatically for the clinical movement, and the start position or theend position may be included in the path region. For example, todetermine whether the movement was within the path region may includedetermining whether the movement started in the start position or endedin the end position. The display 510 may be used by a therapist (e.g.,on a therapist user interface) to modify the path region, the startposition, or the end position. The processor 502 may be used to create avideo or animation using the path region and the information about theclinical movement. For example, the video may include the path regionsuperimposed on captured video or animation of the clinical movement.The video may be played on the display 510. While playing the video onthe display 510, the movement capture apparatus 503 may be used tocapture the movement of the patient. The captured video may be stored inthe video storage 512. The animation may be stored in the animationstorage 514. In an example, a video may be retrieved from the videostorage 512. The retrieved video may include an automatically added pathregion, start position, or end position. In another example, ananimation may be retrieved from the animation storage 514. The retrievedanimation may include an automatically added path region, startposition, or end position.

The processor 502 of the augmented reality device 501 includes anaugmented reality modeler 518. The augmented reality device 501 mayinclude a camera 506. The system 500 may include the database 511, whichmay communicate with the augmented reality device 501.

The processor 502 may identify an object in a real environment, such asthrough processing information received using the camera 506. Forexample, the processor 502 may receive information, such as an image ora series of images from the camera 506 and identify in the image or theseries of images, the object. The processor 502 may create virtualtarget (e.g., a line, a bubble, etc.) in an augmented reality (AR)environment. The virtual target may have a fixed position, such as aposition fixed with respect to the object. For example, the virtualtarget may be positioned in the AR environment such that the virtualtarget remains fixed when the AR device 501 moves. In an example, thevirtual target may be fixed without respect to a view presented to auser of the AR device. In an example, the virtual target may be a sphere(e.g. bubble) represented in the real environment in a position fixedwith respect to the patient. The patient, for example, may be seated,and instructed to do a leg extension (e.g., knee extension from a flexedposition). The sphere may be placed at a final destination for thepatient's foot in performing the knee extension (e.g., in front of thepatient at an approximate distance the length of the patient's leg atapproximately the height of the patient's knee). The sphere maydisappear when the patient's foot enters the sphere (or comes close toit). The disappearance of the sphere may indicate a successfullyperformed movement. In another example, a series of spheres, such asalong the path of the foot from the flexed knee position to the extendedknee position, may be virtually displayed and disappear as the footenters each subsequent sphere (which may overlap to show an intendedpath of movement).

The display 510 may display the AR environment overlaid on the realenvironment. The display 510 may show the virtual target, using the ARdevice 501, in the fixed position in the AR environment. In an example,the display 501 may remove the virtual target from display in the ARenvironment in response to detecting a user interaction with the virtualtarget. For example, when the virtual target is a line or bubble, thevirtual target may be removed (e.g., fade out, popped, explode, etc.)when a user interacts with the virtual target (e.g., kicks the bubble,moves a body part past the line, etc.).

In an example, the camera 506 may be used to identify the object in thereal environment. The camera 506 may send information, such as images tothe processor 502 about the object, and the processor 502 may use theraw information (e.g., raw images) to identify the object in the realenvironment. The augmented reality device 501 may include a sensor 516,such as an infrared sensor. In another example, the sensor may be on theobject. In an example, the processor 502 may receive information fromthe sensor 516 on the object to identify the object. The camera 506 orthe sensor 516 may be used to detect movement that may be interpreted bythe processor 502 as attempted or intended interaction by the user withthe virtual target.

The augmented reality modeler 518 may be used by the processor 502 tocreate the augmented reality environment. For example, the augmentedreality modeler 518 may receive dimensions of a room, such as from thecamera 506 or sensor 516, and create the augmented reality environmentto fit within the physical structure of the room. In another example,physical objects may be present in the room and the augmented realitymodeler 518 may use the physical objects to present virtual objects inthe augmented reality environment. For example, the augmented realitymodeler 518 may use or detect a table present in the room and present avirtual object as resting on the table. In an example, a plurality ofphysical items may be used by a user to interact with the virtualtarget. The virtual object may be used by a user to complete a physicaltherapy movement.

FIG. 6 illustrates a flow chart showing a technique 600 fortelerehabilitation in accordance with some embodiments. The technique600 includes an operation 602 to analyze a clinical movement todetermine a path of motion of the clinical movement. The clinicalmovement may be received from a movement capture apparatus that capturesthe clinical movement, performed by a therapist. To analyze the clinicalmovement may include determining a start position or an end position.

The technique 600 includes an operation 604 to automatically define apath region using the path of motion. To automatically define the pathregion may include defining a start region or an end region, such as byusing the start position or the end position (e.g., the start region maybe an area surrounding the start position, or the end region may be anarea surrounding the end position). The start position or the endposition may be determined from a limb position of the therapist in avideo. In an example, the path region may include a predetermined areasurrounding the path of motion. The technique 600 includes an operation606 to receive information about a movement of a patient along the pathof motion. The information may include whether the movement stays withinthe path of motion, moves outside the path of motion, follows the pathof motion (e.g., within a range of error around the path of motion), orthe like. In an example, receiving information includes capturing andanalyzing video. In an example, receiving information may includeanalyzing live video of the patient, superimposing the live video on theanimation including the path of motion and path region to provideimmediate visual feedback to the patient regarding the exercise/gesturebeing performed.

The technique 600 includes an operation 608 to determine whether themovement was within the path region. To determine whether the movementwas within the path region may include determining whether the movementstarted in the start region or ended in the end region. In an example,determining whether the movement was within the path region includesdetermining that the movement was outside the path region. In anexample, determining whether the movement was within the path regionincludes determining that the movement was within the path region.

In an example, the technique 600 may include monitoring a patientmovement. The patient movement may be compared with the path region todetermine whether the movement was within the path region, whether themovement started in the start region, and whether the movement ended inthe end region. The technique 600 may include sending feedbackcomprising a real-time depiction of the movement of the patient. Thereal-time depiction may include displaying the movement on a display,such as with visual indicators of the start region, the end region, orthe path region. For example, a bubble may be used to represent thestart region. The path region may be represented by a series of bubblesor visual indicators of edges of the path region. In an example, thereal-time depiction of the patient movement includes an animationrepresenting the clinical movement including the path region. In anotherexample, the real-time depiction includes a video of the clinicalmovement including the path region. The animation or video representingthe clinical movement may include a color-coded visual effect toindicate compliance or non-compliance of the patient movement with theclinical movement. For example, the edges of the path region may turnred in response to detecting that the patient has moved outside the pathregion. The edges may be green when the patient movement is within thepath region. The edges may turn yellow when the movement approacheswithin a specified distance of the edges of the path region. In anexample, the start region and the end region may be represented bybubbles, which may be sustained until the patient moves into the startregion or the end region. When the patient moves into the start regionor the end region, the bubbles may pop. In an example, the path regionmay be represented by a series of bubbles (e.g., starting after thestart region bubble and ending before the end region bubble, along thepath of motion). The patient movement through the series of bubbles maypop the series of bubbles in order while completing the movement.

The technique 600 includes an operation 610 to send feedback indicatingwhether the movement was within the path region. Sending feedback mayinclude providing an alert to the patient or the therapist when themovement was determined to be outside the path region. The feedback mayinclude an indication that the movement is to be repeated. The feedbackmay include an alert that the patient failed to complete the movement.Sending feedback may include providing an indication to the patient orthe therapist that the movement successfully mimicked the clinicalmovement when the movement falls within the path region. The feedbackmay include an indication that the patient completed the movementsuccessfully. The feedback may include visual feedback, auditoryfeedback, haptic feedback, non-contact feedback, or the like. Thefeedback may be presented on a user interface on a display. The userinterface may include an option to modify the path region. The therapistmay select a modification to the path region, the start region, or theend region.

In an example, the technique 600 includes creating a video or animationusing the path region and the information about the clinical movement.The video or animation may include the path region superimposed oncaptured video or animation of the clinical movement. The video oranimation may be played on a display. In an example, the video oranimation may be played on a display while capturing the movement of thepatient using the movement capture apparatus. The movement may bedisplayed on the display.

FIG. 7 illustrates a flow chart showing a technique 700 for displayingdirections related to a therapeutic movement for a patient within anaugmented reality environment in accordance with some embodiments. Thetechnique 700 includes an operation 702 to display the augmented realityenvironment, such as an AR environment overlaid on a real environment.Operation 702 may be performed by an AR device. The AR environment maybe created using an augmented reality modeler. The technique 700includes an operation 704 to identify an object in a real environment.The object may be identified using a camera of an AR device. In anexample, the object may be identified using a sensor on the object toidentify the object.

The technique 700 includes an operation 706 to create a virtual targetin the augmented reality environment. The virtual target may have afixed position relative to the object in the real environment. Thevirtual target may be fixed without respect to a view presented to auser of an augmented reality device. For example, the virtual target mayremain in a position fixed in the real environment when an AR devicemoves. The virtual target may be used, for example, by a user tocomplete a physical therapy movement. Displaying the virtual target inthe fixed position may include displaying a bubble, such as in aposition at an end of a physical therapy movement to be completed by auser. The bubble may be popped by displaying an animation, in responseto user interaction with the bubble (e.g., completion of the therapeuticmovement). In an example, displaying the virtual target in the fixedposition includes displaying a virtual physical therapy movement using avirtual avatar, the virtual physical therapy movement corresponding tothe physical therapy movement to be completed by the user. In anexample, the virtual avatar is a virtual representation of a physicaltherapist.

The technique 700 includes an operation 708 to display the virtualtarget in the augmented reality environment. The virtual target may bedisplayed in the fixed position for use in the therapeutic movement inthe augmented reality environment by the augmented reality device. Thefixed position may be located at an intended starting, ending, orintermediate location of the therapeutic movement. The technique 700includes an operation 710 to remove the virtual target from display inthe augmented reality environment. Operation 710 may include removingthe virtual target from display in response to detecting a userinteraction with the virtual target, wherein the user indication mayindicate completion of the therapeutic movement. To detect the userinteraction with the virtual target may include using a camera or asensor (e.g., a camera or a sensor on an AR device).

The technique 700 may include displaying in the AR environment aplurality of physical items to be used by a user to interact with thevirtual target. In an example, the plurality of physical items may beidentified in the AR environment, such as with a visual indicator overor near a physical item, or by displaying a virtual representation ofthe physical item. The technique 700 may include displaying anindication in the augmented reality environment, the indicationincluding clinical information regarding the user interaction.

FIG. 8 illustrates a flow chart showing a technique for displayingaugmented reality clinical movements in accordance with someembodiments. The technique 800 includes an operation 802 to analyzeinformation about a clinical movement, such as a clinical movement by atherapist, to determine a path of motion representative of at least aportion of the clinical movement. In an example, a clinical movement mayinclude a movement for an exercise, a routine, a stretch, anoccupational therapy movement, a physical therapy movement, or the like.The clinical movement of the therapist may be captured using a movementcapture apparatus. In an example, the movement capture apparatusincludes an infrared sensor and the path of motion is determined from aseries of snapshots, the snapshots including infrared sensor data fromthe infrared sensor.

The technique 800 includes an operation 804 to automatically define apath region and a virtual target in an augmented reality environmentoverlaid on a real environment using the path of motion. The virtualtarget may have a fixed position relative to an object in the realenvironment. For example, the virtual target may have a fixed positionrelative to aspects of a room (e.g., a floor, a wall, a ceiling, etc.),fixtures (e.g., a table, a chair, etc.), moving objects (e.g., a person,a pet, a skateboard, etc.), or the like. In an example, the path regionis defined as a region including a specified distance around the path ofmotion. Operation 804 may include receiving a modification to the pathregion from the therapist on a user interface of a display.

The technique 800 includes an operation 806 to display the path regionand the virtual target at the fixed position in the augmented realityenvironment. The path region and the virtual target may be displayedusing an augmented reality device. In an example, the fixed position maybe located at an intended ending location of the path region.

In an example, displaying the virtual target includes displaying amovement task object representative of a real-world object used inoccupational or physical therapy. For example, the real-world object mayinclude dishes, and a movement task associated with the dishes mayinclude display a virtual representation of the dishes to be “moved”virtually in the augmented reality environment to mimic the real-worldtask of putting the dishes away. In another example, the real-worldobject may be a pet (e.g., a cat or dog), and a real-world task that maybe simulated in the augmented reality environment may include steppingover the pet. In yet another example, a box may be augmented to appearas a stair, and the task may include stepping onto the box to simulateclimbing a stair. Other examples may include other real-world objectsfor avoidance, everyday tasks, or the like, for tasks such as steppingon a gas/brake pedal, lifting a cup of coffee, taking a picture, typing,brushing teeth, opening a door, getting into a car, etc.

The augmented reality device may be used to display, in the augmentedreality environment, a plurality of physical items to be used in theuser interaction with the virtual target. For example, a virtualrepresentation of a weight may be displayed, the weight corresponding toa weight to be lifted in the clinical movement. In another example, aresistance band may be shown, including for example a color, for use inthe clinical movement.

In an example, the technique 800 may include displaying an indication inthe augmented reality environment, the indication including clinicalinformation regarding a user interaction. For example, the indicationmay be displayed to the user to indicate that the clinical movement wasperformed successfully or was not completed. Information displayed tothe user may include encouragement or advice (e.g., “lift leg just alittle bit more”). In another example, the indication may be displayedto the therapist to update the therapist on a patient's technique,success, failure, progress, exertion level, etc.

While the virtual target is displayed, the user may interact with thevirtual target in the augmented reality environment. The user's actionin the real environment may trigger an effect in the augmented realityenvironment. For example, the virtual target may be removed from displayin the augmented reality environment in response to detecting a userinteraction with the virtual target, such as completion of the clinicalmovement. Removing the virtual target may indicate completion of theclinical movement. In an example, the user interaction with the virtualtarget may be detected using a camera, the user interaction causing thevirtual target to be removed from the augmented reality environment. Forexample, the camera and the augmented reality display device maycommunicate (or communicate with a third device) to determine whetherthe displayed virtual target overlaid in the real environment and theuser interaction in the real environment occupy overlapping space in thereal environment.

In an example, the technique 800 includes sending feedback comprising areal-time depiction of movement along the path region toward the virtualtarget corresponding to a patient movement attempting to reproduce theclinical movement using the augmented reality device. In an example, thereal-time depiction of the patient movement includes an animationrepresenting the clinical movement including the path region. Forexample, the animation representing the clinical movement may becolor-coded to indicate compliance or non-compliance of the patientmovement with the clinical movement.

In an example, the technique 800 includes receiving information about amovement of a user along the path of motion to the virtual target. Theinformation may be used to determine whether the movement was within thepath region, such as based on analysis of the information about themovement of the user. In response to determining that the movement waswithin the path region, the technique 800 may include displaying, usingthe augmented reality device, feedback indicating that the movement waswithin the path region.

In an example, an AR device may be used by a surgeon or a patientpostoperatively. For example, the surgeon or the patient may use the ARdevice to view aspects of an implant or a surgery, such as using animplant sensor, medical imaging (e.g., x-ray, MRI, CT-scan, etc.), orimages or video taken during surgery. The AR device may display storedvideo of the implant or aspects of the surgery postoperatively as athree-dimensional virtual object overlaid on a real environment. Thevirtual object may be viewed by the surgeon or the patient (or otheruser), in three-dimensions such that the virtual object appears to be ina room with the surgeon or the patient and the virtual object may beviewed from multiple different angles as the surgeon or the patientmoves the AR device around the virtual object. For example, the virtualobject may be stationary or may move with one or more points fixed to anaspect of the real environment (e.g., the virtual object may appear tobe resting on a table). In an example, the surgeon and the patient mayview the virtual object together, and the surgeon may control thevirtual object, to show the virtual object moving or may point outcertain aspects of the virtual object.

FIG. 9 illustrates generally an example of a block diagram of a machine900 upon which any one or more of the techniques (e.g., methodologies)discussed herein may perform in accordance with some embodiments. Inalternative embodiments, the machine 900 may operate as a standalonedevice or may be connected (e.g., networked) to other machines. In anetworked deployment, the machine 900 may operate in the capacity of aserver machine, a client machine, or both in server-client networkenvironments. The machine 900 may be a personal computer (PC), a tabletPC, a personal digital assistant (PDA), a mobile telephone, a webappliance, a network router, switch or bridge, or any machine capable ofexecuting instructions (sequential or otherwise) that specify actions tobe taken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein, such as cloud computing, software as aservice (SaaS), other computer cluster configurations.

Examples, as described herein, may include, or may operate on, logic ora number of components, modules, or like mechanisms. Such mechanisms aretangible entities (e.g., hardware) capable of performing specifiedoperations when operating. In an example, the hardware may bespecifically configured to carry out a specific operation (e.g.,hardwired). In an example, the hardware may include configurableexecution units (e.g., transistors, circuits, etc.) and a computerreadable medium containing instructions, where the instructionsconfigure the execution units to carry out a specific operation when inoperation. The configuring may occur under the direction of theexecutions units or a loading mechanism. Accordingly, the executionunits are communicatively coupled to the computer readable medium whenthe device is operating. For example, under operation, the executionunits may be configured by a first set of instructions to implement afirst set of features at one point in time and reconfigured by a secondset of instructions to implement a second set of features.

Machine (e.g., computer system) 900 may include a hardware processor 902(e.g., a central processing unit (CPU), a graphics processing unit(GPU), a hardware processor core, or any combination thereof), a mainmemory 904 and a static memory 906, some or all of which may communicatewith each other via an interlink (e.g., bus) 908. The machine 900 mayfurther include a display unit 910, an alphanumeric input device 912(e.g., a keyboard), and a user interface (UI) navigation device 914(e.g., a mouse). In an example, the display unit 910, alphanumeric inputdevice 912 and UI navigation device 914 may be a touch screen display.The display unit 910 may include goggles, glasses, or other AR or VRdisplay components. For example, the display unit may be worn on a headof a user and may provide a heads-up-display to the user. Thealphanumeric input device 912 may include a virtual keyboard (e.g., akeyboard displayed virtually in a VR or AR setting.

The machine 900 may additionally include a storage device (e.g., driveunit) 916, a signal generation device 918 (e.g., a speaker), a networkinterface device 920, and one or more sensors 921, such as a globalpositioning system (GPS) sensor, compass, accelerometer, or othersensor. The machine 900 may include an output controller 928, such as aserial (e.g., universal serial bus (USB), parallel, or other wired orwireless (e.g., infrared (IR), near field communication (NFC), etc.)connection to communicate or control one or more peripheral devices.

The storage device 916 may include a machine readable medium 922 that isnon-transitory on which is stored one or more sets of data structures orinstructions 924 (e.g., software) embodying or utilized by any one ormore of the techniques or functions described herein. The instructions924 may also reside, completely or at least partially, within the mainmemory 904, within static memory 906, or within the hardware processor902 during execution thereof by the machine 900. In an example, one orany combination of the hardware processor 902, the main memory 904, thestatic memory 906, or the storage device 916 may constitute machinereadable media.

While the machine readable medium 922 is illustrated as a single medium,the term “machine readable medium” may include a single medium ormultiple media (e.g., a centralized or distributed database, orassociated caches and servers) configured to store the one or moreinstructions 924.

The term “machine readable medium” may include any medium that iscapable of storing, encoding, or carrying instructions for execution bythe machine 900 and that cause the machine 900 to perform any one ormore of the techniques of the present disclosure, or that is capable ofstoring, encoding or carrying data structures used by or associated withsuch instructions. Non-limiting machine readable medium examples mayinclude solid-state memories, and optical and magnetic media. Specificexamples of machine readable media may include: non-volatile memory,such as semiconductor memory devices (e.g., Electrically ProgrammableRead-Only Memory (EPROM), Electrically Erasable Programmable Read-OnlyMemory (EEPROM)) and flash memory devices; magnetic disks, such asinternal hard disks and removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks.

The instructions 924 may further be transmitted or received over acommunications network 926 using a transmission medium via the networkinterface device 920 utilizing any one of a number of transfer protocols(e.g., frame relay, internet protocol (IP), transmission controlprotocol (TCP), user datagram protocol (UDP), hypertext transferprotocol (HTTP), etc.). Example communication networks may include alocal area network (LAN), a wide area network (WAN), a packet datanetwork (e.g., the Internet), mobile telephone networks (e.g., cellularnetworks), Plain Old Telephone (POTS) networks, and wireless datanetworks (e.g., Institute of Electrical and Electronics Engineers (IEEE)802.11 family of standards known as Wi-Fi®, as the personal area networkfamily of standards known as Bluetooth® that are promulgated by theBluetooth Special Interest Group, peer-to-peer (P2P) networks, amongothers. In an example, the network interface device 920 may include oneor more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or oneor more antennas to connect to the communications network 926. In anexample, the network interface device 920 may include a plurality ofantennas to wirelessly communicate using at least one of single-inputmultiple-output (SIMO), multiple-input multiple-output (MIMO), ormultiple-input single-output (MISO) techniques. The term “transmissionmedium” shall be taken to include any intangible medium that is capableof storing, encoding or carrying instructions for execution by themachine 900, and includes digital or analog communications signals orother intangible medium to facilitate communication of such software.

VARIOUS NOTES & EXAMPLES

Each of these non-limiting examples may stand on its own, or may becombined in various permutations or combinations with one or more of theother examples.

Example 1 is a method for displaying augmented reality clinicalmovements, the method comprising: analyzing information about a clinicalmovement of a therapist, captured using a movement capture apparatus, todetermine a path of motion representative of at least a portion of theclinical movement; automatically defining a path region and a virtualtarget in an augmented reality environment overlaid on a realenvironment using the path of motion, the virtual target having a fixedposition relative to an object in the real environment; and displaying,using an augmented reality device, the path region and the virtualtarget at the fixed position in the augmented reality environment, thefixed position located at an intended ending location of the pathregion.

In Example 2, the subject matter of Example 1 optionally includesreceiving information about a movement of a user along the path ofmotion to the virtual target.

In Example 3, the subject matter of Example 2 optionally includesdetermining, based on analysis of the information about the movement ofthe user, whether the movement was within the path region, and inresponse to determining that the movement was within the path region,displaying, using the augmented reality device, feedback indicating thatthe movement was within the path region.

In Example 4, the subject matter of any one or more of Examples 1-3optionally include removing the virtual target from display in theaugmented reality environment, in response to detecting a userinteraction with the virtual target indicating completion of a usermovement replicating the clinical movement.

In Example 5, the subject matter of any one or more of Examples 1-4optionally include wherein the movement capture apparatus includes aninfrared sensor and the path of motion is determined from a series ofsnapshots, the snapshots including infrared sensor data from theinfrared sensor.

In Example 6, the subject matter of Example 5 optionally includeswherein the path region is defined as a region including a specifieddistance around the path of motion.

In Example 7, the subject matter of any one or more of Examples 1-6optionally include receiving a modification to the path region from thetherapist on a user interface of a display.

In Example 8, the subject matter of any one or more of Examples 1-7optionally include sending feedback comprising a real-time depiction ofmovement along the path region toward the virtual target correspondingto a patient movement attempting to reproduce the clinical movementusing the augmented reality device.

In Example 9, the subject matter of Example 8 optionally includeswherein the real-time depiction of the patient movement includes ananimation representing the clinical movement including the path region.

In Example 10, the subject matter of Example 9 optionally includeswherein the animation representing the clinical movement is color-codedto indicate compliance or non-compliance of the patient movement withthe clinical movement.

In Example 11, the subject matter of any one or more of Examples 1-10optionally include displaying, in the augmented reality environment, aplurality of physical items to be used in the user interaction with thevirtual target.

In Example 12, the subject matter of any one or more of Examples 1-11optionally include detecting a user interaction with the virtual targetin data received from a camera, the user interaction causing the virtualtarget to be removed from the augmented reality environment.

In Example 13, the subject matter of any one or more of Examples 1-12optionally include wherein displaying the virtual target includesdisplaying a movement task object representative of a real-world objectused in occupational therapy.

In Example 14, the subject matter of any one or more of Examples 1-13optionally include displaying an indication in the augmented realityenvironment, the indication including clinical information regarding auser interaction.

Example 15 is an augmented reality device for displaying directionsrelated to the clinical movement for a patient within an augmentedreality environment comprising: a processor to: analyze informationabout a clinical movement of a therapist, captured using a movementcapture apparatus, to determine a path of motion representative of atleast a portion of the clinical movement; and automatically define apath region and a virtual target in an augmented reality environmentoverlaid on a real environment using the path of motion, the virtualtarget having a fixed position relative to an object in the realenvironment; and a display to: display, using an augmented realitydevice, the path region and the virtual target at the fixed position inthe augmented reality environment, the fixed position located at anintended ending location of the path region.

In Example 16, the subject matter of Example 15 optionally includeswherein the processor is further to receive information about a movementof a user along the path of motion to the virtual target.

In Example 17, the subject matter of Example 16 optionally includeswherein the processor is further to determine, based on analysis of theinformation about the movement of the user, whether the movement waswithin the path region, and in response to determining that the movementwas within the path region, the display is further to display, using theaugmented reality device, feedback indicating that the movement waswithin the path region.

In Example 18, the subject matter of any one or more of Examples 15-17optionally include wherein the display is further to remove the virtualtarget from display in the augmented reality environment, in response todetecting a user interaction with the virtual target indicatingcompletion of a user movement replicating the clinical movement.

In Example 19, the subject matter of any one or more of Examples 15-18optionally include wherein the movement capture apparatus furtherincludes an infrared sensor and the path of motion is determined from aseries of snapshots, the snapshots including infrared sensor data fromthe infrared sensor.

In Example 20, the subject matter of Example 19 optionally includeswherein the path region is defined as a region including a specifieddistance around the path of motion.

In Example 21, the subject matter of any one or more of Examples 15-20optionally include wherein the processor is further to receive amodification to the path region from the therapist on a user interfaceof the display.

In Example 22, the subject matter of any one or more of Examples 15-21optionally include wherein the processor is further to generate feedbackcomprising a real-time depiction of movement along the path regiontoward the virtual target corresponding to a patient movement attemptingto reproduce the clinical movement using the augmented reality device.

In Example 23, the subject matter of Example 22 optionally includeswherein the real-time depiction of the patient movement includes ananimation representing the clinical movement including the path region.

In Example 24, the subject matter of Example 23 optionally includeswherein the animation representing the clinical movement is color-codedto indicate compliance or non-compliance of the patient movement withthe clinical movement.

In Example 25, the subject matter of any one or more of Examples 15-24optionally include wherein the display is further to display, in theaugmented reality environment, a plurality of physical items to be usedin the user interaction with the virtual target.

In Example 26, the subject matter of any one or more of Examples 15-25optionally include wherein the processor is further to detect a userinteraction with the virtual target in data received from a camera, theuser interaction causing the virtual target to be removed from theaugmented reality environment.

In Example 27, the subject matter of any one or more of Examples 15-26optionally include wherein to display the virtual target, the display isfurther to display a movement task object representative of a real-worldobject used in occupational therapy.

In Example 28, the subject matter of any one or more of Examples 15-27optionally include wherein the display is further to display anindication in the augmented reality environment, the indicationincluding clinical information regarding a user interaction.

Example 29 is a system comprising: a movement capture apparatusincluding a camera to capture information about a clinical movement of atherapist; and a processor to: analyze the information to determine apath of motion representative of at least a portion of the clinicalmovement; and automatically define a path region and a virtual target inan augmented reality environment overlaid on a real environment usingthe path of motion, the virtual target having a fixed position relativeto an object in the real environment; and an augmented reality displaydevice to display the path region and the virtual target at the fixedposition in the augmented reality environment, the fixed positionlocated at an intended ending location of the path region.

In Example 30, the subject matter of Example 29 optionally includes acamera to capture information about a movement of a user along the pathof motion to the virtual target.

In Example 31, the subject matter of Example 30 optionally includeswherein the processor is further to determine, based on analysis of theinformation about the movement of the user, whether the movement waswithin the path region, and in response to determining that the movementwas within the path region, the augmented reality display device isfurther to display feedback indicating that the movement was within thepath region.

In Example 32, the subject matter of any one or more of Examples 29-31optionally include wherein the augmented reality display device isfurther to remove the virtual target from display in the augmentedreality environment, in response to detecting a user interaction withthe virtual target indicating completion of a user movement replicatingthe clinical movement.

In Example 33, the subject matter of any one or more of Examples 29-32optionally include wherein the movement capture apparatus furtherincludes an infrared sensor and the path of motion is determined from aseries of snapshots, the snapshots including infrared sensor data fromthe infrared sensor.

In Example 34, the subject matter of Example 33 optionally includeswherein the path region is defined as a region including a specifieddistance around the path of motion.

In Example 35, the subject matter of any one or more of Examples 29-34optionally include wherein the processor is further to receive amodification to the path region from the therapist on a user interfaceof the augmented reality display device.

In Example 36, the subject matter of any one or more of Examples 29-35optionally include wherein the processor is further to generate feedbackcomprising a real-time depiction of movement along the path regiontoward the virtual target corresponding to a patient movement attemptingto reproduce the clinical movement using the augmented reality device.

In Example 37, the subject matter of Example 36 optionally includeswherein the real-time depiction of the patient movement includes ananimation representing the clinical movement including the path region.

In Example 38, the subject matter of Example 37 optionally includeswherein the animation representing the clinical movement is color-codedto indicate compliance or non-compliance of the patient movement withthe clinical movement.

In Example 39, the subject matter of any one or more of Examples 29-38optionally include wherein the augmented reality display device isfurther to display, in the augmented reality environment, a plurality ofphysical items to be used in the user interaction with the virtualtarget.

In Example 40, the subject matter of any one or more of Examples 29-39optionally include wherein the processor is further to detect a userinteraction with the virtual target in data received from a camera, theuser interaction causing the virtual target to be removed from theaugmented reality environment.

In Example 41, the subject matter of any one or more of Examples 29-40optionally include wherein to display the virtual target, the augmentedreality display device is further to display a movement task objectrepresentative of a real-world object used in occupational therapy.

In Example 42, the subject matter of any one or more of Examples 29-41optionally include wherein the augmented reality display device isfurther to display an indication in the augmented reality environment,the indication including clinical information regarding a userinteraction.

Example 43 is at least one machine-readable medium includinginstructions for displaying augmented reality clinical movements, whichwhen executed by a machine, cause the machine to: analyze informationabout a clinical movement of a therapist, captured using a movementcapture apparatus, to determine a path of motion representative of atleast a portion of the clinical movement; automatically define a pathregion and a virtual target in an augmented reality environment overlaidon a real environment using the path of motion, the virtual targethaving a fixed position relative to an object in the real environment;and display, using an augmented reality device, the path region and thevirtual target at the fixed position in the augmented realityenvironment, the fixed position located at an intended ending locationof the path region.

In Example 44, the subject matter of Example 43 optionally includesinstructions to receive information about a movement of a user along thepath of motion to the virtual target.

In Example 45, the subject matter of Example 44 optionally includesinstructions to determine, based on analysis of the information aboutthe movement of the user, whether the movement was within the pathregion, and in response to determining that the movement was within thepath region, display, using the augmented reality device, feedbackindicating that the movement was within the path region.

In Example 46, the subject matter of any one or more of Examples 43-45optionally include instructions to remove the virtual target fromdisplay in the augmented reality environment, in response to detecting auser interaction with the virtual target indicating completion of a usermovement replicating the clinical movement.

In Example 47, the subject matter of any one or more of Examples 43-46optionally include wherein the movement capture apparatus includes aninfrared sensor and the path of motion is determined from a series ofsnapshots, the snapshots including infrared sensor data from theinfrared sensor.

In Example 48, the subject matter of Example 47 optionally includeswherein the path region is defined as a region including a specifieddistance around the path of motion.

In Example 49, the subject matter of any one or more of Examples 43-48optionally include instructions to receive a modification to the pathregion from the therapist on a user interface of a display.

In Example 50, the subject matter of any one or more of Examples 43-49optionally include instructions to generate feedback comprising areal-time depiction of movement along the path region toward the virtualtarget corresponding to a patient movement attempting to reproduce theclinical movement using the augmented reality device.

In Example 51, the subject matter of Example 50 optionally includeswherein the real-time depiction of the patient movement includes ananimation representing the clinical movement including the path region.

In Example 52, the subject matter of Example 51 optionally includeswherein the animation representing the clinical movement is color-codedto indicate compliance or non-compliance of the patient movement withthe clinical movement.

In Example 53, the subject matter of any one or more of Examples 43-52optionally include instructions to display, in the augmented realityenvironment, a plurality of physical items to be used in the userinteraction with the virtual target.

In Example 54, the subject matter of any one or more of Examples 43-53optionally include instructions to detect a user interaction with thevirtual target in data received from a camera, the user interactioncausing the virtual target to be removed from the augmented realityenvironment.

In Example 55, the subject matter of any one or more of Examples 43-54optionally include wherein the instructions to display the virtualtarget include instructions to display a movement task objectrepresentative of a real-world object used in occupational therapy.

In Example 56, the subject matter of any one or more of Examples 43-55optionally include instructions to display an indication in theaugmented reality environment, the indication including clinicalinformation regarding a user interaction.

Example 57 is a method for telerehabilitation, the method comprising:receiving information about a clinical movement of a therapist capturedusing a movement capture apparatus; analyzing the clinical movement todetermine a path of motion representative of at least a portion of theclinical movement; automatically defining a path region using the pathof motion; receiving information about a movement of a patient along thepath of motion; determining, based on analysis of the information aboutthe movement of the patient, whether the movement was within the pathregion; and sending feedback indicating whether the movement was withinthe path region.

In Example 58, the subject matter of Example 57 optionally includeswherein analyzing the clinical movement includes determining a startposition and an end position of the clinical movement.

In Example 59, the subject matter of Example 58 optionally includeswherein automatically defining the path region includes automaticallydefining a start region and an end region using the start position andthe end position.

In Example 60, the subject matter of Example 59 optionally includeswherein determining whether the movement was within the path regionincludes determining whether the movement started in the start regionand ended in the end region.

In Example 61, the subject matter of any one or more of Examples 57-60optionally include wherein the movement capture apparatus includes aninfrared sensor and the path of motion is determined from a series ofsnapshots, the snapshots including infrared sensor data from theinfrared sensor.

In Example 62, the subject matter of Example 61 optionally includeswherein the path region is defined as a region including a specifieddistance around the path of motion.

In Example 63, the subject matter of any one or more of Examples 57-62optionally include wherein the path region includes a predetermined areasurrounding the path of motion.

In Example 64, the subject matter of any one or more of Examples 57-63optionally include wherein determining whether the movement was withinthe path region includes determining that the movement was outside thepath region.

In Example 65, the subject matter of Example 64 optionally includeswherein sending the feedback includes providing an alert to the patientthat the movement was outside the path region and that the movement isto be repeated.

In Example 66, the subject matter of any one or more of Examples 64-65optionally include wherein sending the feedback includes providing analert to the therapist that the patient failed to complete the movement.

In Example 67, the subject matter of any one or more of Examples 57-66optionally include wherein determining whether the movement was withinthe path region includes determining that the movement was within thepath region.

In Example 68, the subject matter of Example 67 optionally includeswherein sending the feedback includes providing an indication to thepatient that the movement successfully mimicked the clinical movement.

In Example 69, the subject matter of any one or more of Examples 67-68optionally include wherein sending the feedback includes providing anindication to the therapist that the patient completed the movementsuccessfully.

Example 70 is at least one machine-readable medium includinginstructions for receiving information, which when executed by amachine, cause the machine to: receive information about a clinicalmovement of a therapist captured using a movement capture apparatus;analyze the clinical movement to determine a path of motion of theclinical movement; automatically define a path region using the path ofmotion; receive information about a movement of a patient along the pathof motion; determine whether the movement was within the path region;and send feedback indicating whether the movement was within the pathregion.

In Example 71, the subject matter of Example 70 optionally includeswherein the feedback is visual, auditory, or haptic.

In Example 72, the subject matter of any one or more of Examples 70-71optionally include instructions to receive a modification to the pathregion from the therapist on a user interface of a display.

In Example 73, the subject matter of any one or more of Examples 70-72optionally include instructions to create a video using the path regionand the information about the clinical movement, the video including thepath region superimposed on captured video of the clinical movement.

In Example 74, the subject matter of Example 73 optionally includesinstructions to play the video on a display while capturing the movementof the patient using the movement capture apparatus.

Example 75 is a system comprising: a movement capture apparatus; memory;and a processor connected to the memory, the processor to: receiveinformation about a clinical movement performed by a therapist andcaptured using the movement capture apparatus; analyze the clinicalmovement to determine a path of motion of the clinical movement;automatically define a path region using the path of motion; receiveinformation about a movement of a patient along the path of motion, theinformation generated by the movement capture apparatus; determinewhether the movement was within the path region; and send feedbackindicating whether the movement was within the path region.

In Example 76, the subject matter of Example 75 optionally includeswherein to automatically define the path region, the processor isfurther to automatically define a start region and an end region using astart position and an end position.

In Example 77, the subject matter of Example 76 optionally includeswherein to determine whether the movement was within the path region,the processor is further to determine whether the movement started inthe start region and ended in the end region.

In Example 78, the subject matter of any one or more of Examples 75-77optionally include wherein the processor is to generate feedbackcomprising a real-time depiction of the movement of the patient.

In Example 79, the subject matter of Example 78 optionally includeswherein the real-time depiction of the patient movement includes ananimation representing the clinical movement including the path region.

In Example 80, the subject matter of any one or more of Examples 78-79optionally include wherein the animation representing the clinicalmovement is color-coded to indicate compliance or non-compliance of thepatient movement with the clinical movement.

Example 81 is a method for displaying directions related to atherapeutic movement for a patient within an augmented realityenvironment, the method comprising: displaying the augmented realityenvironment overlaid on a real environment using an augmented realitydevice; identifying an object in the real environment; creating avirtual target in the augmented reality environment with a fixedposition relative to the object; displaying, using the augmented realitydevice, the virtual target in the fixed position for use in thetherapeutic movement in the augmented reality environment, the fixedposition located at an intended ending location of the therapeuticmovement; and removing the virtual target from display in the augmentedreality environment, in response to detecting a user interaction withthe virtual target indicating completion of a user movement replicatingthe therapeutic movement.

In Example 82, the subject matter of Example 81 optionally includesdisplaying, in the augmented reality environment, a plurality ofphysical items to be used in the user interaction with the virtualtarget.

In Example 83, the subject matter of any one or more of Examples 81-82optionally include wherein displaying the virtual target in the fixedposition includes displaying a bubble in the intended ending location.

In Example 84, the subject matter of Example 83 optionally includeswherein removing the virtual target includes displaying an animationpopping the bubble.

In Example 85, the subject matter of any one or more of Examples 81-84optionally include wherein the virtual target is fixed relative to theobject and without respect to a view presented to a user of theaugmented reality device.

In Example 86, the subject matter of any one or more of Examples 81-85optionally include wherein identifying the object in the realenvironment includes using a camera of the augmented reality device.

In Example 87, the subject matter of any one or more of Examples 81-86optionally include wherein identifying the object in the realenvironment includes using a sensor on the object to identify theobject.

In Example 88, the subject matter of any one or more of Examples 81-87optionally include using an augmented reality modeler to create theaugmented reality environment.

In Example 89, the subject matter of any one or more of Examples 81-88optionally include wherein detecting the user interaction with thevirtual target includes using a camera.

In Example 90, the subject matter of any one or more of Examples 81-89optionally include wherein displaying the virtual target in the fixedposition includes displaying a virtual physical therapy movement using avirtual avatar, the virtual physical therapy movement corresponding tothe physical therapy movement to be completed by the user.

In Example 91, the subject matter of Example 90 optionally includeswherein the virtual avatar is a virtual representation of a physicaltherapist.

In Example 92, the subject matter of any one or more of Examples 81-91optionally include displaying an indication in the augmented realityenvironment, the indication including clinical information regarding theuser interaction.

Example 93 is an augmented reality device for displaying directionsrelated to a therapeutic movement for a patient within an augmentedreality environment, the augmented reality device comprising: aprocessor to: identify an object in a real environment; and create avirtual target in an augmented reality environment with a fixed positionrelative to the object; and a display to: display the augmented realityenvironment overlaid on the real environment; display, using theaugmented reality device, the virtual target in the fixed position foruse in the therapeutic movement in the augmented reality environment,the fixed position located at an intended ending location of thetherapeutic movement; and remove the virtual target from display in theaugmented reality environment, in response to detecting a userinteraction with the virtual target indicating completion of a usermovement replicating the therapeutic movement.

In Example 94, the subject matter of Example 93 optionally includeswherein the virtual target is fixed without respect to a view presentedto a user of the augmented reality device.

In Example 95, the subject matter of any one or more of Examples 93-94optionally include a camera to identify the object in the realenvironment.

In Example 96, the subject matter of any one or more of Examples 93-95optionally include wherein to identify the object in the realenvironment, the processor is further to receive information from asensor on the object to identify the object.

In Example 97, the subject matter of any one or more of Examples 93-96optionally include wherein the processor is further to use an augmentedreality modeler to create the augmented reality environment.

In Example 98, the subject matter of any one or more of Examples 93-97optionally include a camera to detect the user interaction with thevirtual target.

In Example 99, the subject matter of any one or more of Examples 93-98optionally include wherein the display is further to display, in theaugmented reality environment, a plurality of physical items to be usedby a user to interact with the virtual target.

In Example 100, the subject matter of any one or more of Examples 93-99optionally include wherein the virtual target is used by a user tocomplete a physical therapy movement.

In Example 101, the subject matter of any one or more of Examples 1-99may optionally include a technique or system including displaycomponents to allow a therapist to select an object and place the objectin an AR environment.

In Example 102, the subject matter of Example 101 may optionally includegenerating feedback when a user interaction with the placed object isdetected.

In Example 103, the subject matter of any one or more of Examples101-102 optionally include wherein detecting a user interaction with theplaced object includes receiving data from a camera, the userinteraction causing the placed object to be removed from the ARenvironment.

Method examples described herein may be machine or computer-implementedat least in part. Some examples may include a computer-readable mediumor machine-readable medium encoded with instructions operable toconfigure an electronic device to perform methods as described in theabove examples. An implementation of such methods may include code, suchas microcode, assembly language code, a higher-level language code, orthe like. Such code may include computer readable instructions forperforming various methods. The code may form portions of computerprogram products. Further, in an example, the code may be tangiblystored on one or more volatile, non-transitory, or non-volatile tangiblecomputer-readable media, such as during execution or at other times.Examples of these tangible computer-readable media may include, but arenot limited to, hard disks, removable magnetic disks, removable opticaldisks (e.g., compact disks and digital video disks), magnetic cassettes,memory cards or sticks, random access memories (RAMs), read onlymemories (ROMs), and the like.

What is claimed is:
 1. A method for evaluating clinical movements, themethod comprising: receiving clinical movement data indicative of aclinical movement of a therapist, the clinical movement data capturedusing a movement capture apparatus; analyzing the clinical movement datato determine a three-dimensional path of motion representative of aportion of the clinical movement; automatically defining athree-dimensional path region using the three-dimensional path ofmotion; creating an animation of the portion of the clinical movementincluding a visual representation of the three-dimensional path region;displaying the animation including the visual representation of thethree-dimensional path region; receiving patient movement dataindicative of a movement of a patient along the three-dimensional pathof motion; determining, based on analysis of the patient movement dataabout the movement of the patient, whether the movement was within thethree-dimensional path region; and sending feedback indicating whetherthe movement was within the three-dimensional path region.
 2. The methodof claim 1, wherein analyzing the clinical movement data includesdetermining a start position and an end position of the clinicalmovement, wherein automatically defining the three-dimensional pathregion includes automatically defining a start region and an end regionusing the start position and the end position, and wherein determiningwhether the movement was within the three-dimensional path regionincludes determining whether the movement started in the start regionand ended in the end region.
 3. The method of claim 1, wherein receivingthe clinical movement data includes receiving a plurality of snapshotsfrom the movement capture apparatus that includes an infrared sensor;and wherein analyzing the clinical movement data to determine thethree-dimensional path of motion analyzing the plurality of snapshotsincluding infrared sensor data from the infrared sensor.
 4. The methodof claim 1, wherein defining the three-dimensional path region includesdefining a specified distance around the three-dimensional path ofmotion.
 5. The method of claim 1, wherein determining whether themovement was within the three-dimensional path region includesdetermining that the movement was outside the three-dimensional pathregion.
 6. The method of claim 5, wherein sending the feedback includesproviding an alert to the patient that the movement was outside thethree-dimensional path region and that the movement is to be repeated.7. The method of claim 5, wherein sending the feedback includesproviding an alert to the therapist that the patient failed to completethe movement.
 8. The method of claim 1, wherein determining whether themovement was within the three-dimensional path region includesdetermining that the movement was within the three-dimensional, pathregion, and wherein sending the feedback includes providing anindication to the patient that the movement successfully mimicked theclinical movement or providing an indication to the therapist that thepatient completed the movement successfully.
 9. The method of claim 1,further comprising receiving a modification to the three-dimensionalpath region from the therapist on a user interface of a display.
 10. Themethod of claim 1, wherein the visual representation of thethree-dimensional path region is color-coded within the animation of theportion of the clinical movement to indicate compliance ornon-compliance of the movement of the patient with the clinicalmovement.
 11. At least one machine-readable medium includinginstructions for receiving information, which when executed by amachine, cause the machine to: receive clinical movement data indicativeof a clinical movement of a therapist, the clinical movement datacaptured using a movement capture apparatus; analyze the clinicalmovement data to determine a three-dimensional path of motionrepresentative of a portion of the clinical movement; automaticallydefine a three-dimensional path region using the three-dimensional pathof motion; create an animation of the portion of the clinical movementincluding a visual representation of the three-dimensional path region;cause the animation to be displayed including the visual representationof the three-dimensional path region; receive patient movement dataindicative of a movement of a patient along the three-dimensional pathof motion; determine, based on analysis of the patient movement dataabout the movement of the patient, whether the movement was within thethree-dimensional path region; and send feedback indicating whether themovement was within the three-dimensional path region.
 12. The at leastone machine-readable medium of claim 11, wherein to analyze the clinicalmovement data, the instructions are further to cause the machine todetermine a start position and an end position of the clinical movement,wherein to automatically define the three-dimensional path region, theinstructions are further to cause the machine to automatically define astart region and an end region using the start position and the endposition, and wherein to determine whether the movement was within thethree-dimensional path region, the instructions are further to cause themachine to determining whether the movement started in the start regionand ended in the end region.
 13. The at least one machine-readablemedium of claim 11, wherein to define the three-dimensional path region,the instructions are further to cause the machine to define a specifieddistance around the three-dimensional path of motion.
 14. The at leastone machine-readable medium of claim 11, wherein the visualrepresentation of the three-dimensional path region is color-codedwithin the animation of the portion of the clinical movement to indicatecompliance or non-compliance of the movement of the patient with theclinical movement.
 15. The at least one machine-readable medium of claim11, wherein to send the feedback, the instructions are further to causethe machine to provide an alert to the patient that the movement wasoutside the three-dimensional path region and that the movement is to berepeated or provide an alert to the therapist that the patient failed tocomplete the movement.
 16. A system comprising: a display device;memory; and a processor connected to the memory, the processor to:receive clinical movement data indicative of a clinical movement of atherapist, the clinical movement data captured using a movement captureapparatus; analyze the clinical movement data to determine athree-dimensional path of motion representative of a portion of theclinical movement; automatically define a three-dimensional path regionusing the three-dimensional path of motion; create an animation of theportion of the clinical movement including a visual representation ofthe three-dimensional path region; send the animation to be displayed onthe display device, including the visual representation of thethree-dimensional path region; receive patient movement data indicativeof a movement of a patient along the three-dimensional path of motion;determine, based on analysis of the patient movement data the movementof the patient, whether the movement was within the three-dimensionalpath region; and send feedback indicating whether the movement waswithin the three-dimensional path region.
 17. The system of claim 16,wherein to determine whether the movement was within thethree-dimensional path region, the processor is further to determinethat the movement was within the three-dimensional path region, andwherein to send the feedback, the processor is further to provide anindication to the patient that the movement successfully mimicked theclinical movement or provide an indication to the therapist that thepatient completed the movement successfully.
 18. The system of claim 16,wherein the processor is further to receive a modification to thethree-dimensional path region from the therapist on a user interface ofa display.
 19. The system of claim 16, wherein the visual representationof the three-dimensional path region is color-coded within the animationof the portion of the clinical movement to indicate compliance ornon-compliance of the movement of the patient with the clinicalmovement.
 20. The system of claim 16, wherein the system furthercomprises the movement capture apparatus, including an infrared sensor,the infrared sensor to capture a series of snapshots including infraredsensor data to be used to determine the three-dimensional path ofmotion.